Validation and application of the Non-Verbal Behavior Analyzer: an automated tool to assess non-verbal emotional expressions in psychotherapy

Background Emotions play a key role in psychotherapy. However, a problem with examining emotional states via self-report questionnaires is that the assessment usually takes place after the actual emotion has been experienced which might lead to biases and continuous human ratings are time and cost intensive. Using the AI-based software package Non-Verbal Behavior Analyzer (NOVA), video-based emotion recognition of arousal and valence can be applied in naturalistic psychotherapeutic settings. In this study, four emotion recognition models (ERM) each based on specific feature sets (facial: OpenFace, OpenFace-Aureg; body: OpenPose-Activation, OpenPose-Energy) were developed and compared in their ability to predict arousal and valence scores correlated to PANAS emotion scores and processes of change (interpersonal experience, coping experience, affective experience) as well as symptoms (depression and anxiety in HSCL-11). Materials and methods A total of 183 patient therapy videos were divided into a training sample (55 patients), a test sample (50 patients), and a holdout sample (78 patients). The best ERM was selected for further analyses. Then, ERM based arousal and valence scores were correlated with patient and therapist estimates of emotions and processes of change. Furthermore, using regression models arousal and valence were examined as predictors of symptom severity in depression and anxiety. Results The ERM based on OpenFace produced the best agreement to the human coder rating. Arousal and valence correlated significantly with therapists’ ratings of sadness, shame, anxiety, and relaxation, but not with the patient ratings of their own emotions. Furthermore, a significant negative correlation indicates that negative valence was associated with higher affective experience. Negative valence was found to significantly predict higher anxiety but not depression scores. Conclusion This study shows that emotion recognition with NOVA can be used to generate ERMs associated with patient emotions, affective experiences and symptoms. Nevertheless, limitations were obvious. It seems necessary to improve the ERMs using larger databases of sessions and the validity of ERMs needs to be further investigated in different samples and different applications. Furthermore, future research should take ERMs to identify emotional synchrony between patient and therapists into account

MultiMediate '22: Backchannel Detection and Agreement Estimation in Group Interactions

Backchannels, i.e. short interjections of the listener, serve important meta-conversational purposes like signifying attention or indicating agreement. Despite their key role, automatic analysis of backchannels in group interactions has been largely neglected so far. The MultiMediate challenge addresses, for the first time, the tasks of backchannel detection and agreement estimation from backchannels in group conversations. This paper describes the MultiMediate challenge and presents a novel set of annotations consisting of 7234 backchannel instances for the MPIIGroupInteraction dataset. Each backchannel was additionally annotated with the extent by which it expresses agreement towards the current speaker. In addition to a an analysis of the collected annotations, we present baseline results for both challenge tasks.Comment: ACM Multimedia 202

Prazosin during threat discrimination boosts memory of the safe stimulus

The α-1 adrenoreceptor antagonist prazosin has shown promise in the treatment of post-traumatic stress disorder (PTSD) symptoms, but its mechanisms are not well understood. Here we administered prazosin or placebo prior to threat conditioning (day 1) and tested subsequent extinction (day 2) and reextinction (day 3) in healthy human participants. Prazosin did not affect threat conditioning but augmented stimulus discrimination during extinction and reextinction, via lower responding to the safe stimulus. These results suggest that prazosin during threat acquisition may have influenced encoding or consolidation of safety processing in particular, subsequently leading to enhanced discrimination between the safe and threatening stimuli

Evidence for a minimal role of stimulus awareness in reversal of threat learning

In an ever-changing environment, survival depends on learning which stimuli represent threat, and also on updating such associations when circumstances shift. It has been claimed that humans can acquire physiological responses to threat-associated stimuli even when they are unaware of them, but the role of awareness in updating threat contingencies remains unknown. This complex process-generating novel responses while suppressing learned ones-relies on distinct neural mechanisms from initial learning, and has only been shown with awareness. Can it occur unconsciously? Here, we present evidence that threat reversal may not require awareness. Participants underwent classical threat conditioning to visual stimuli that were suppressed from awareness. One of two images was paired with an electric shock; halfway through the experiment, contingencies were reversed and the shock was paired with the other image. Despite variations in suppression across participants, we found that physiological responses reflected changes in stimulus-threat pairings independently of stimulus awareness. These findings suggest that unconscious affective processing may be sufficiently flexible to adapt to changing circumstances

Towards an Intelligent Tutor for Mathematical Proofs

Computer-supported learning is an increasingly important form of study since it allows for independent learning and individualized instruction. In this paper, we discuss a novel approach to developing an intelligent tutoring system for teaching textbook-style mathematical proofs. We characterize the particularities of the domain and discuss common ITS design models. Our approach is motivated by phenomena found in a corpus of tutorial dialogs that were collected in a Wizard-of-Oz experiment. We show how an intelligent tutor for textbook-style mathematical proofs can be built on top of an adapted assertion-level proof assistant by reusing representations and proof search strategies originally developed for automated and interactive theorem proving. The resulting prototype was successfully evaluated on a corpus of tutorial dialogs and yields good results.Comment: In Proceedings THedu'11, arXiv:1202.453

Highly resolved observations of trace gases in the lowermost stratosphere and upper troposphere from the Spurt project: an overview

During SPURT (Spurenstofftransport in der Tropopausenregion, trace gas transport in the tropopause region) we performed measurements of a wide range of trace gases with different lifetimes and sink/source characteristics in the northern hemispheric upper troposphere (UT) and lowermost stratosphere (LMS). A large number of in-situ instruments were deployed on board a Learjet 35A, flying at altitudes up to 13.7 km, at times reaching to nearly 380 K potential temperature. Eight measurement campaigns (consisting of a total of 36 flights), distributed over all seasons and typically covering latitudes between 35° N and 75° N in the European longitude sector (10° W–20° E), were performed. Here we present an overview of the project, describing the instrumentation, the encountered meteorological situations during the campaigns and the data set available from SPURT. Measurements were obtained for N2O, CH4, CO, CO2, CFC12, H2, SF6, NO, NOy, O3 and H2O. We illustrate the strength of this new data set by showing mean distributions of the mixing ratios of selected trace gases, using a potential temperature – equivalent latitude coordinate system. The observations reveal that the LMS is most stratospheric in character during spring, with the highest mixing ratios of O3 and NOy and the lowest mixing ratios of N2O and SF6. The lowest mixing ratios of NOy and O3 are observed during autumn, together with the highest mixing ratios of N2O and SF6 indicating a strong tropospheric influence. For H2O, however, the maximum concentrations in the LMS are found during summer, suggesting unique (temperature- and convection-controlled) conditions for this molecule during transport across the tropopause. The SPURT data set is presently the most accurate and complete data set for many trace species in the LMS, and its main value is the simultaneous measurement of a suite of trace gases having different lifetimes and physical-chemical histories. It is thus very well suited for studies of atmospheric transport, for model validation, and for investigations of seasonal changes in the UT/LMS, as demonstrated in accompanying and elsewhere published studies

Highly resolved observations of trace gases in the lowermost stratosphere and upper troposphere from the Spurt project: an overview

During SPURT (Spurenstofftransport in der Tropopausenregion, trace gas transport in the tropopause region) we performed measurements of a wide range of trace gases with different lifetimes and sink/source characteristics in the northern hemispheric upper troposphere (UT) and lowermost stratosphere (LMS). A large number of in-situ instruments were deployed on board a Learjet 35A, flying at altitudes up to 13.7 km, at times reaching to nearly 380 K potential temperature. Eight measurement campaigns (consisting of a total of 36 flights), distributed over all seasons and typically covering latitudes between 35° N and 75° N in the European longitude sector (10° W–20° E), were performed. Here we present an overview of the project, describing the instrumentation, the encountered meteorological situations during the campaigns and the data set available from SPURT. Measurements were obtained for N2O, CH4, CO, CO2, CFC12, H2, SF6, NO, NOy, O3 and H2O. We illustrate the strength of this new data set by showing mean distributions of the mixing ratios of selected trace gases, using a potential temperature – equivalent latitude coordinate system. The observations reveal that the LMS is most stratospheric in character during spring, with the highest mixing ratios of O3 and NOy and the lowest mixing ratios of N2O and SF6. The lowest mixing ratios of NOy and O3 are observed during autumn, together with the highest mixing ratios of N2O and SF6 indicating a strong tropospheric influence. For H2O, however, the maximum concentrations in the LMS are found during summer, suggesting unique (temperature- and convection-controlled) conditions for this molecule during transport across the tropopause. The SPURT data set is presently the most accurate and complete data set for many trace species in the LMS, and its main value is the simultaneous measurement of a suite of trace gases having different lifetimes and physical-chemical histories. It is thus very well suited for studies of atmospheric transport, for model validation, and for investigations of seasonal changes in the UT/LMS, as demonstrated in accompanying and elsewhere published studies

MultiMediate'23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions

Automatic analysis of human behaviour is a fundamental prerequisite for the creation of machines that can effectively interact with- and support humans in social interactions. In MultiMediate'23, we address two key human social behaviour analysis tasks for the first time in a controlled challenge: engagement estimation and bodily behaviour recognition in social interactions. This paper describes the MultiMediate'23 challenge and presents novel sets of annotations for both tasks. For engagement estimation we collected novel annotations on the NOvice eXpert Interaction (NOXI) database. For bodily behaviour recognition, we annotated test recordings of the MPIIGroupInteraction corpus with the BBSI annotation scheme. In addition, we present baseline results for both challenge tasks.Comment: ACM MultiMedia'2

Therapeutic T cells induce tumor-directed chemotaxis of innate immune cells through tumor-specific secretion of chemokines and stimulation of B16BL6 melanoma to secrete chemokines

Background: The mechanisms by which tumor-specific T cells induce regression of established metastases are not fully characterized. In using the poorly immunogenic B16BL6-D5 (D5) melanoma model we reported that T cell-mediated tumor regression can occur independently of perforin, IFN-gamma or the combination of both. Characterization of regressing pulmonary metastases identified macrophages as a major component of the cells infiltrating the tumor after adoptive transfer of effector T cells. This led us to hypothesize that macrophages played a central role in tumor regression following T-cell transfer. Here, we sought to determine the factors responsible for the infiltration of macrophages at the tumor site. Methods: These studies used the poorly immunogenic D5 melanoma model. Tumor-specific effector T cells, generated from tumor vaccine-draining lymph nodes (TVDLN), were used for adoptive immunotherapy and in vitro analysis of chemokine expression. Cellular infiltrates into pulmonary metastases were determined by immunohistochemistry. Chemokine expression by the D5 melanoma following co-culture with T cells, IFN-gamma or TNF-alpha was determined by RT-PCR and ELISA. Functional activity of chemokines was confirmed using a macrophage migration assay. T cell activation of macrophages to release nitric oxide (NO) was determined using GRIES reagent. Results: We observed that tumor-specific T cells with a type 1 cytokine profile also expressed message for and secreted RANTES, MIP-1 alpha and MIP-1 beta following stimulation with specific tumor. Unexpectedly, D5 melanoma cells cultured with IFN-gamma or TNF-alpha, two type 1 cytokines expressed by therapeutic T cells, secreted Keratinocyte Chemoattractant (KC), MCP-1, IP-10 and RANTES and expressed mRNA for MIG. The chemokines released by T cells and cytokine-stimulated tumor cells were functional and induced migration of the DJ2PM macrophage cell line. Additionally, tumor-specific stimulation of wt or perforin-deficient (PKO) effector T cells induced macrophages to secrete nitric oxide (NO), providing an additional effector mechanism for T cell-mediated tumor regression. Conclusion: These data suggest two possible sources for chemokine secretion that stimulates macrophage recruitment to the site of tumor metastases. Both appear to be initiated by T cell recognition of specific antigen, but one is dependent on the tumor cells to produce the chemokines that recruit macrophages

Simulating rewetting events in intermittent rivers and ephemeral streams: a global analysis of leached nutrients and organic matter

Climate change and human pressures are changing the global distribution and extent of intermittent rivers and ephemeral streams (IRES), which comprise half of the global river network area. IRES are characterized by periods of flow cessation, during which channel substrates accumulate and undergo physico‐chemical changes (preconditioning), and periods of flow resumption, when these substrates are rewetted and release pulses of dissolved nutrients and organic matter (OM). However, there are no estimates of the amounts and quality of leached substances, nor is there information on the underlying environmental constraints operating at the global scale. We experimentally simulated, under standard laboratory conditions, rewetting of leaves, riverbed sediments, and epilithic biofilms collected during the dry phase across 205 IRES from five major climate zones. We determined the amounts and qualitative characteristics of the leached nutrients and OM, and estimated their areal fluxes from riverbeds. In addition, we evaluated the variance in leachate characteristics in relation to selected environmental variables and substrate characteristics. We found that sediments, due to their large quantities within riverbeds, contribute most to the overall flux of dissolved substances during rewetting events (56‐98%), and that flux rates distinctly differ among climate zones. Dissolved organic carbon, phenolics, and nitrate contributed most to the areal fluxes. The largest amounts of leached substances were found in the continental climate zone, coinciding with the lowest potential bioavailability of the leached organic matter. The opposite pattern was found in the arid zone. Environmental variables expected to be modified under climate change (i.e. potential evapotranspiration, aridity, dry period duration, land use) were correlated with the amount of leached substances, with the strongest relationship found for sediments. These results show that the role of IRES should be accounted for in global biogeochemical cycles, especially because prevalence of IRES will increase due to increasing severity of drying events